Real-time identification of data candidates for classification based compression

ABSTRACT

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate for determining if the data candidate may benefit from the classification-based compression, wherein a ratio is summed between the actual number of the characters and the expected number of the characters, and then dividing the ratio by a number of the data classes that are not empty, wherein the non-classifiable data are included in the number of the data classes during the dividing, and the number of the data classes, that are not empty, have characters that belong to the class that were observed in the input; and the classification-based compression is performed on the data candidates if the ratio exceeds a threshold.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.14/074,053, filed on Nov. 7, 2013, which is a Continuation of U.S.patent application Ser. No. 13/738,262, filed on Jan. 10, 2013, both ofwhich are hereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates in general to computers, and moreparticularly to real-time identification of data candidates forclassification-based compression in a computing environment.

DESCRIPTION OF THE RELATED ART

In today's society, computer systems are commonplace. Computer systemsmay be found in the workplace, at home, or at school. Computer systemsmay include data storage systems, or disk storage systems, to processand store data. Data storage systems, or disk storage systems, areutilized to process and store data. A storage system may include one ormore disk drives. These data processing systems typically require alarge amount of data storage. Customer data, or data generated by userswithin the data processing system, occupies a great portion of this datastorage. Many of these computer systems include virtual storagecomponents.

Column based compression, classification compression, and datacompression is widely used to reduce the amount of data required toprocess, transmit, or store a given quantity of information. Datacompression is the coding of data to minimize its representation.Compression can be used, for example, to reduce the storage requirementsfor files, to increase the communication rate over a channel, or toreduce redundancy prior to encryption for greater security.

SUMMARY OF THE DESCRIBED EMBODIMENTS

Various embodiments are provided for identification of data candidatesfor data processing performed in real time by a processor device in acomputing environment. In one embodiment, the method comprises samplingdata candidates for performing a classification-based compression uponthe data candidates. A heuristic is computed on a randomly selected datasample from the data candidate for determining if the data candidate maybenefit from the classification-based compression, wherein a ratio issummed between the actual number of the characters and the expectednumber of the characters, and then dividing the ratio by a number of thedata classes that are not empty, wherein the non-classifiable data areincluded in the number of the data classes during the dividing, and thenumber of the data classes, that are not empty, have characters thatbelong to the class that were observed in the input; and theclassification-based compression is performed on the data candidates ifthe ratio exceeds a threshold.

In addition to the foregoing exemplary method embodiment, otherexemplary system and computer product embodiments are provided andsupply related advantages. The foregoing summary has been provided tointroduce a selection of concepts in a simplified form that are furtherdescribed below in the Detailed Description. This Summary is notintended to identify key features or essential features of the claimedsubject matter, nor is it intended to be used as an aid in determiningthe scope of the claimed subject matter. The claimed subject matter isnot limited to implementations that solve any or all disadvantages notedin the background.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict embodiments of the invention and are not therefore to beconsidered to be limiting of its scope, the invention will be describedand explained with additional specificity and detail through the use ofthe accompanying drawings, in which:

FIG. 1 is a block diagram illustrating a computer storage environmenthaving an exemplary storage device in which aspects of the presentinvention may be realized;

FIG. 2 is a block diagram illustrating a hardware structure of anexemplary data storage system in a computer system in which aspects ofthe present invention may be realized;

FIG. 3 is a flow chart diagram illustrating an exemplary method forreal-time identification of data candidates for classification-basedcompression in which aspects of the present invention may be realized;and

FIG. 4 is a flow chart diagram illustrating an additional exemplarymethod for identifying and sampling data candidates for performing aclassification-based compression in in which aspects of the presentinvention may be realized.

DETAILED DESCRIPTION OF THE DRAWINGS

As previously mentioned, computing systems are used to store and managea variety of types of data. Compressing similar data using the samecompression stream improves the compression ratio and reduces therequired storage. For data that have an internal structure, e.g.,tabular data or semi-structured data, separating sequences of data thatbelong to a class of data into separate compression streams improves thecompression ratio. For example, when data is composed of interleavingsequences of text and numbers, compressing the text and numbersseparately will provide a better compression ratio. A well-knowntechnology for databases is column compression, where data stored indatabase tables is compressed column-wise (not row-wise), hence thenotation of column compression, proving a better compression ratio, asdata in a column is typically more similar than data between columnsallowing the compression module to better exploit similarities andprovide a better data reduction. The concept may be adapted tosemi-structured data such as Hypertext Markup Language (HTML),Extensible Markup Language (XML), JavaScript Objected Notation (JSON),etc. By understanding the underlying structure of the data, each classof data may be identified and compressed—that is by usingclassification-based compression. Classification-based compression is ageneralization of column compression, where the structure of the data isnot strictly well-defined columns. In classification-based compression,the data of similar type is grouped together in the same compressionstream. Classification-based compression allows for a smaller alphabetfootprint, and assists in identifying repetitions that are furtherapart. These groups (e.g., the data of similar type that is groupedtogether in the same compression stream) can be called “virtual columns”in an analogy for column compression. However, not all data isstructured or semi-structured, therefore, performingclassification-based compression should be done on data which have aclear tabular or semi-tabular structure. For example, images, videos andencrypted or compressed data have no such tabular like structure.

In a block storage system, data blocks are written to the storagewithout any indication what type of data is written, and no indicationof relation between writes is given. Thus, it is impossible to usecolumn compression, as the columns need to be given in advances, but themore general method of classification-based compression can be used.Classification-based compression is best employed on data which isstructured or semi-structured. Therefore, it is necessary to identifywhich writes include data with a tabular-like structure, and whichwrites do not.

Thus, in one embodiment, the present invention provides a solution forreal-time identification as to whether a write buffer is potentiallycomposed of data sequences (e.g., candidates) that can be separated intodifferent compression streams. Since the “type” of data written is notprovided (to the storage system), and present invention allows for thestorage system to decide in an autonomic manner how to compress thedata. A detection operation reads small fragments (e.g., random samplesfrom a data stream and/or data block) of the input data that arerandomly selected, and estimates if classes of data are present in theinput, and then provide a decision whether to use classification-basedcompression on this analyzed data. When the data classes are defined inadvance, the presence of the class is determined by analyzing how manycharacters (bytes) from the input “falls” into the predefined dataclasses. When the data classes are not defined in advance, the classesare determined automatically by looking at pairs of characters anddetermining character clusters. Thus, embodiments are provided for whenthe classes of data are defined in advance and when the data classesshould be determined in real-time. The identification of classes in thedata may be performed in a single pass, is not limited to a rigidformat, and supports both fixed and variable data lengths.

As will be described below, in one embodiment, data candidates (e.g., awrite operation in the block storage system) are identified in real timefor performing a classification-based compression, performed by aprocessor device, in a computing environment. Data candidates aresampled for performing a classification-based compression upon the datacandidates. A heuristic is computed on a randomly selected data samplefrom the data candidate for determining if the data candidate maybenefit from the classification-based compression. A decision isprovided for approving the classification-based compression on the datacandidates according to the heuristic.

Turning now to FIG. 1, exemplary architecture 10 of data storage systemsin a computing environment is depicted. The computer system 10 includescentral processing unit (CPU) 12, which is connected to mass storagedevice(s) 14 and memory device 16. Mass storage devices can include harddisk drive (HDD) devices, solid-state devices (SSD) etc., which can beconfigured in a redundant array of independent disks (RAID). The backupoperations further described can be executed on device(s) 14, located insystem 10 or elsewhere. Memory device 16 can include such memory aselectrically erasable programmable read only memory (EEPROM) or a hostof related devices. Memory device 16 and mass storage device 14 areconnected to CPU 12 via a signal-bearing medium. In addition, CPU 12 isconnected through communication port 18 to a communication network 20,having an attached plurality of additional computer systems 22 and 24.

FIG. 2 is an exemplary block diagram 200 showing a hardware structure ofa data storage system in a computer system according to the presentinvention. Referring to FIG. 2, there are shown host computers 210, 220,225, each acting as a central processing unit for performing dataprocessing a part of a data storage system 200. The hosts (physical orvirtual devices), 210, 220, and 225 may be one or more new physicaldevices or logical devices to accomplish the purposes of the presentinvention in the data storage system 200. In one embodiment, by way ofexample only, a data storage system 200 may be implemented as IBM®System Storage™ DS8000™. A network connection 260 may be a fibre channelfabric, a fibre channel point to point link, a fibre channel overethernet fabric or point to point link, a FICON or ESCON I/O interface,any other I/O interface type, a wireless network, a wired network, aLAN, a WAN, heterogeneous, homogeneous, public (i.e. the Internet),private, or any combination thereof. The hosts, 210, 220, and 225 may belocal or distributed among one or more locations and may be equippedwith any type of fabric (or fabric channel) (not shown in FIG. 2) ornetwork adapter 260 to the storage controller 240, such as Fibrechannel, FICON, ESCON, Ethernet, fiber optic, wireless, or coaxialadapters. Data storage system 200 is accordingly equipped with asuitable fabric (not shown in FIG. 2) or network adapter 260 tocommunicate. Data storage system 200 is depicted in FIG. 1 comprisingstorage controller 240 and storage 230.

To facilitate a clearer understanding of the methods described herein,storage controller 240 is shown in FIG. 2 as a single processing unit,including a microprocessor 242, system memory 243 and nonvolatilestorage (“NVS”) 216, which will be described in more detail below. It isnoted that in some embodiments, storage controller 240 is comprised ofmultiple processing units, each with their own processor complex andsystem memory, and interconnected by a dedicated network within datastorage system 200. Storage 230 may be comprised of one or more storagedevices, such as storage arrays, which are connected to storagecontroller 240 by a storage network.

In some embodiments, the devices included in storage 230 may beconnected in a loop architecture. Storage controller 240 manages storage230 and facilitates the processing of write and read requests intendedfor storage 230. The system memory 243 of storage controller 240 storesprogram instructions and data, which the processor 242 may access forexecuting functions and method steps associated with managing storage230 and executing the steps and methods of the present invention in acomputer storage environment. In one embodiment, system memory 243includes, is associated, or is in communication with the operationsoftware 250 in a computer storage environment, including the methodsand operations described herein. As shown in FIG. 2, system memory 243may also include or be in communication with a cache 245 for storage230, also referred to herein as a “cache memory”, for buffering “writedata” and “read data”, which respectively refer to write/read requestsand their associated data. In one embodiment, cache 245 is allocated ina device external to system memory 243, yet remains accessible bymicroprocessor 242 and may serve to provide additional security againstdata loss, in addition to carrying out the operations as described inherein.

In some embodiments, cache 245 is implemented with a volatile memory andnon-volatile memory and coupled to microprocessor 242 via a local bus(not shown in FIG. 2) for enhanced performance of data storage system200. The NVS 216 included in data storage controller is accessible bymicroprocessor 242 and serves to provide additional support foroperations and execution of the present invention as described in otherfigures. The NVS 216, may also referred to as a “persistent” cache, or“cache memory” and is implemented with nonvolatile memory that may ormay not utilize external power to retain data stored therein. The NVSmay be stored in and with the Cache 245 for any purposes suited toaccomplish the objectives of the present invention. In some embodiments,a backup power source (not shown in FIG. 2), such a battery, suppliesNVS 216 with sufficient power to retain the data stored therein in caseof power loss to data storage system 200. In certain embodiments, thecapacity of NVS 216 is less than or equal to the total capacity of cache245.

Storage 230 may be physically comprised of one or more storage devices,such as storage arrays. A storage array is a logical grouping ofindividual storage devices, such as a hard disk. In certain embodiments,storage 230 is comprised of a JBOD (Just a Bunch of Disks) array or aRAID (Redundant Array of Independent Disks) array. A collection ofphysical storage arrays may be further combined to form a rank, whichdissociates the physical storage from the logical configuration. Thestorage space in a rank may be allocated into logical volumes, whichdefine the storage location specified in a write/read request.

In one embodiment, by way of example only, the storage system as shownin FIG. 2 may include a logical volume, or simply “volume,” may havedifferent kinds of allocations. Storage 230 a, 230 b and 230 n are shownas ranks in data storage system 200, and are referred to herein as rank230 a, 230 b and 230 n. Ranks may be local to data storage system 200,or may be located at a physically remote location. In other words, alocal storage controller may connect with a remote storage controllerand manage storage at the remote location. Rank 230 a is shownconfigured with two entire volumes, 234 and 236, as well as one partialvolume 232 a. Rank 230 b is shown with another partial volume 232 b.Thus volume 232 is allocated across ranks 230 a and 230 b. Rank 230 n isshown as being fully allocated to volume 238—that is, rank 230 n refersto the entire physical storage for volume 238. From the above examples,it will be appreciated that a rank may be configured to include one ormore partial and/or entire volumes. Volumes and ranks may further bedivided into so-called “tracks,” which represent a fixed block ofstorage. A track is therefore associated with a given volume and may begiven a given rank.

The storage controller 240 may include a classification-basedcompression module 255, an identification module 257, and a data classmodule 259 in a computer storage environment. The classification-basedcompression module 255, the identification module 257, and the dataclass module 259 may work in conjunction with each and every componentof the storage controller 240, the hosts 210, 220, 225, and storagedevices 230. The classification-based compression module 255, theidentification module 257, and the data class module 259 may bestructurally one complete module working together and in conjunctionwith each other for performing such functionality as described below, ormay be individual modules. The classification-based compression module255, the identification module 257, and the data class module 259 mayalso be located in the cache 245 or other components of the storagecontroller 240 to accomplish the purposes of the present invention.

The storage controller 240 may be constructed with a control switch 241for controlling the fiber channel protocol to the host computers 210,220, 225, a microprocessor 242 for controlling all the storagecontroller 240, a nonvolatile control memory 243 for storing amicroprogram (operation software) 250 for controlling the operation ofstorage controller 240, data for control and each table described later,cache 245 for temporarily storing (buffering) data, and buffers 244 forassisting the cache 245 to read and write data, a control switch 241 forcontrolling a protocol to control data transfer to or from the storagedevices 230, classification-based compression module 255, theidentification module 257, and the data class module 259 on whichinformation may be set. Multiple buffers 244 may be implemented with thepresent invention in a computing environment, or performing otherfunctionality in accordance with the mechanisms of the illustratedembodiments.

In one embodiment, by way of example only, the host computers or one ormore physical or virtual devices, 210, 220, 225 and the storagecontroller 240 are connected through a network adaptor (this could be afiber channel) 260 as an interface i.e., via a switch sometimes referredto as “fabric.” In one embodiment, by way of example only, the operationof the system shown in FIG. 2 will be described. The microprocessor 242may control the memory 243 to store command information from the hostdevice (physical or virtual) 210 and information for identifying thehost device (physical or virtual) 210. The control switch 241, thebuffers 244, the cache 245, the operating software 250, themicroprocessor 242, memory 243, NVS 216, classification-basedcompression module 255, the identification module 257, and the dataclass module 259 are in communication with each other and may beseparate or one individual component(s). Also, several, if not all ofthe components, such as the operation software 245 may be included withthe memory 243 in a computer storage environment. Each of the componentswithin the storage device may be linked together and may be incommunication with each other for purposes suited to the presentinvention.

Turning now to FIG. 3, an exemplary method 300 for real-timeidentification of data candidates for classification-based compressionis illustrated. The method 300 begins (step 302) by sampling datacandidates for performing a classification-based compression upon thedata candidates (step 304). A heuristic is computed on a randomlyselected data sample from the data candidate for determining if the datacandidate may benefit from the classification-based compression (step306). A decision is provided for approving the classification-basedcompression on the data candidates according to the heuristic (step308). The method 300 ends (step 310).

Based upon the foregoing, turning now to FIG. 4, an additional exemplarymethod 400 for identifying and sampling data candidates for performing aclassification-based compression is depicted. For sampling datacandidates and calculating the heuristic on a randomly selected datasample from the data candidate for determining if the data candidate maybenefit from the classification-based compression, as described in FIG.3, the method 400 begins (step 402) by estimating if data classes arepresent in the data candidates from a randomly selected data sample (seeFIG. 3 step 304) (step 404). The method determines if the data classesare known in advance (step 406). If yes, the method 400, for each one ofthe data classes, calculates a number of classes that are present in thedata candidates and/or an expected number of characters to be in a dataclass (step 408). An expected number of characters that will not belongto a predefined set of the data classes are calculated (this is thenon-classifiable data) (step 410). An actual number of the charactersfor each of the data classes and the non-classifiable data are alsocalculated (step 412). A minimum, a maximum, an average, and adistribution of a number of consecutive characters in each of the dataclasses are calculated. A ratio, between the actual number of thecharacters and the expected number of the characters, is summed, andthen the ratio is divided by the number of the data classes that are notempty (e.g., characters that belong to the class that were observed inthe input) (step 414). The non-classifiable data are included in thenumber of the data classes during the dividing. The classification-basedcompression is performed on the data candidates if the ratio exceeds athreshold (step 416). However, returning to step 406, if data classesare not known in advance, the method 400 calculates a pair-wisehistogram by counting a number of appearances for each two-consecutivecharacters for creating a graph representation, with each charactersbeing represented in the graph by a node. Each two-consecutivecharacters are represented by an edge between the character nodes, andthe weight associated with the edge is the number of appearances of thetwo-consecutive characters (step 418). A clustering operation is run onthe graph representation whereby the clustering operation computes setsof characters that are tightly grouped and form a group, and ignore(remove) sequences (e.g., candidates) that are not tightly connected,which is a filtering outliers operation. Each group (cluster) of nodes(characters) in the graph representation is a data class (step 420). Themethod 400 then checks to see if more than one cluster is identified(step 422). The heuristic computed for the data classes that are knownin advance for performing the classification-based compression is thenexecuted if more than one cluster is identified (step 424). If theclustering operation outputs a single cluster, the classification-basedcompression is not performed if the clustering operation outputs asingle cluster (step 426). The method 400 ends (step 428).

As described in FIG. 3-4, the embodiments of the present invention, (1)select a small sample of the input data buffer, (2) compute a heuristicto determine whether the input data can benefit fromclassification-based compression, and (3) compresses the input usingclassification-based compression only if it will provide a benefit,otherwise, a default system action (standard compression or nocompression) may be performed. The input data buffer can be anapplication file or a data block. The data sample may be the whole inputbuffer and/or a randomly selected (or predefined) sequences of bytesfrom the buffer. Moreover, when the data classes are known in advance(e.g., uppercase letters, lowercase letters, numbers), the followingheuristic can be used, as illustrated in FIG. 4. 1) For each data class,based on the size of the class (distinct characters in the class), theexpected number of input characters (from the data sample) to be in thedata class is computed. 2) The expected number of input characters thatwill not belong to the pre-defined set of data classes—denoted bynon-classified, is computed. 3) The observed (actual) number of bytesfor each data class, including the non-classified, is computed.Moreover, the minimum, maximum, average, and distribution of the numberof consecutive bytes in each class may be computed. 4) A ratio, betweenobserved and expected number of characters, is summed, and then theratio is divided by the number of classes (the non-classified isconsider as class here) that are not empty (characters that belong tothe class were observed in the input). 5) If the ratio is higher than athreshold, such as two (the value of 2 being used only as an example forillustration purposes), then the data should be compressed usingclassification-based compression. It should be noted that theembodiments described herein are compatible with real-time requirementsof storage array and provide the decision and identification withoutactually performing and using any compression method, reading or parsingthe data, but rather using the sampling and heuristic method foridentification as described above. Moreover, the illustrated embodimentsdo not assume anything on the input buffer block, and provide theidentification by analyzing a sample of the data content without theneed for any known structural format. Also, the embodiments describedherein analyze data blocks by retrieving a sample of the data and doesnot process all the data nor not try to determine the most appropriatecompression method to be used, but rather just to determine theintra-block classification of the data.

According to the forgoing discussion of identifying table boundaries indata block compression, compressing each column independently willprovide a higher compression ratio rather than compressing the entiretable with one stream. It should be noted that a variety of compressiontechniques may be used to accomplish the mechanisms of the presentinvention (e.g., column compression). The reason for the increasedcompression ratio is that the data in each column is relativelyhomogeneous. As a result, efficiency and productivity is increased basedupon the mechanisms of the present invention.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While one or more embodiments of the present invention have beenillustrated in detail, the skilled artisan will appreciate thatmodifications and adaptations to those embodiments may be made withoutdeparting from the scope of the present invention as set forth in thefollowing claims.

What is claimed is:
 1. A method for real-time identification of datacandidates for data processing by a processor device in a computingenvironment, the method comprising: sampling data candidates forperforming a classification-based compression upon the data candidates;and computing a heuristic on a randomly selected data sample from thedata candidate thereby determining if the data candidates may benefitfrom the classification-based compression, wherein a decision isprovided for approving the classification-based compression on the datacandidates according to the heuristic; wherein if data classes areestimated by the processor to be present in the data candidates; summinga ratio between the actual number of the characters and the expectednumber of the characters, and then dividing the ratio by a number of thedata classes that are not empty, wherein the non-classifiable data areincluded in the number of the data classes during the dividing, and thenumber of the data classes, that are not empty, have characters thatbelong to the class that were observed in the input; and performing theclassification-based compression on the data candidates if the ratioexceeds a threshold.
 2. The method of claim 1, further including, ifdata classes are known in advance, computing the heuristic by: for eachone of the data classes, calculating an expected number of characters tobe in a data class, calculating an expected number of characters thatwill not belong to a predefined set of the data classes, wherein thecharacters that do not belong to the predefined set of the data classesare non-classifiable data, and calculating an actual number of thecharacters for each of the data classes and the non-classifiable data,wherein a minimum, a maximum, an average, and a distribution of a numberof consecutive characters in each of the data classes are calculated. 3.The method of claim 1, further including, if data classes are not knownin advance: calculating a pair-wise histogram by counting a number ofappearances for each two-consecutive characters for creating a graphrepresentation having weighted edges, wherein a weight is the number ofappearances of the two-consecutive characters, running a clusteringoperation on the graph representation whereby the clustering operationfilters outliers, and each cluster in the graph representation is a dataclass, and executing the heuristic computed for data classes that areknown in advance for performing the classification-based compression ifmore than one cluster is identified, wherein the classification-basedcompression is not performed if the clustering operation outputs asingle cluster.
 4. The method of claim 1, wherein the identifyingfurther includes: analyzing data in a write buffer independently forspecific character types, and performing the classification-basedcompression if a ratio of the specific character types exceedsnon-specific character types by a threshold amount.
 5. The method ofclaim 1, further including performing a classification identification onthe data candidates during at least one single pass for both fixed datalengths and variable data lengths.
 6. The method of claim 1, wherein thedata candidates are data blocks for determining intra-blockclassification of the data for performing the classification-basedcompression on the data candidates.